Feedback-Based Self-Learning in Large-Scale Conversational AI Agents

Today, most large-scale conversational AI agents (e.g. Alexa, Siri, or Google Assistant) are built using manually annotated data to train the different components of the system. Typically, the accuracy of the ML models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages user-system interaction feedback signals to automate learning without any manual annotation. Users here tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win/loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.Comment: 8 pages, 2 figure

In vivo cyclic loading as a potent stimulatory signal for bone formation inside tissue engineering scaffolds

In clinical situations, bone defects are often located at load bearing sites. Tissue engineering scaffolds are future bone substitutes and hence they will be subjected to mechanical stimulation. The goal of this study was to test if cyclic loading can be used as stimulatory signal for bone formation in a bone scaffold. PLA/ 5% β-TCP scaffolds were implanted in both distal femoral epiphyses of eight rats. Right knees were stimulated (10N, 4Hz, 5min) five times, every two days, starting from the third day after surgery while left knees served as control. Finite element study of the in vivo model showed that the strain applied to the scaffold is similar to physiological strains. Using micro-CT, all knees were scanned five times after the surgery and the related bone parameters of the newly formed bone were quantified. Statistical modeling was used to estimate the evolution of these parameters as a function of time and loading. The results showed that mechanical stimulation had two effects on bone volume (BV): an initial decrease in BV at week 2, and a long-term increase in the rate of bone formation by 28%. At week 13, the BV was then significantly higher in the loaded scaffolds

In vivo assessment of local effects after application of bone screws delivering bisphosphonates into a compromised cancellous bone site

Background: The primary stability of cancellous screw is difficult to obtain in bone of compromised quality and failure of screw fixation is common. To overcome this problem, it is proposed to locally deliver bisphosphonate from the screw An in vivo validation of the increase cancellous screw fixation is then needed in compromised bone. Methods: In this study, we used an overdrilling procedure, which enables consistent modelling of reduced screw stability comparable to compromised cancellous bone. Forty eight adult NZW rabbits were used in this study and all animals underwent bilateral femur implantation. One leg was implanted with the screw containing the bisphosphonate (biocoated group) while the other was used as control (control group) with the screw only. Mechanical testing and micro-CT imaging was used to assess the effect of local drug delivery of Zoledronate on screws fixation at 5 time points. Findings: At the early time points (1, 5, and 10 days), no significant difference could be seen between the biocoated and control groups. At 6 weeks, the bone volume fraction was significantly higher in the trabecular region of the biocoated group. However, this increase did not have a significant effect on the pull-out force. At the last time point, 11 weeks, both the bone volume fraction and the pull-out force were significantly higher in the biocoated group. Interpretation: The results of this study suggest that, in compromised bone, local delivery of bisphosphonate enhances the stability of bone screws

Monitoring bone fetal cells seeded in PLA scaffolds using bioluminescence imaging

In our laboratory, we have shown previously that bone fetal cells enhance osteogenesis in polymeric scaffolds. However, little is known about the longevity of these cells in vivo, and hence their function. A successful tissue engineering approach needs understanding the role osteoinductive cells as well as optimization of their durability in the scaffold. Bioluminescence imaging (BLI) is a potent non-invasive imaging procedure which allows continuous assessing of the behavior of cells in vivo. A collaboration has been made with University Hospital of Geneva, where they have successfully developed a technique for screening myoblasts in vivo using BLI. The goal of this project is to screen the bone fetal cells seeded in polymeric scaffolds in vitro and in vivo using BLI

Biomechanical Aspects of Osteogenesis in Load-Bearing Tissue Engineering Scaffolds

There is increasing evidence that mechanical stimulation affects the fate of tissue engineering scaffolds. In clinical situations, it is critical to accelerate bone formation inside the scaffold to reduce the recovery time and increase the chance of success of the bone substitute. Mechanical stimulation is already employed in bone fractures to enhance healing process and similarly it can be used to accelerate bone formation inside the scaffold. The goal of this thesis is to quantify, through in silico and in vivo methods, how mechanical stimulation enhances bone formation in scaffold and translate the obtained results into clinical applications. In this thesis, a novel mathematical model is proposed to predict the course of bone formation in a scaffold. The mathematical model is identified and further validated using in vivo data. Using rat distal femur as an animal model, a short period of cyclic loading is found to induce long-term effects on bone formation inside scaffold. Mechanical stimulation is shown to increase the stiffness of the bone-scaffold over time. Furthermore, the effect of mechanical stimulation on dynamic of bone formation and resorption is investigated. The surgical preparation of bone-scaffold interface is also shown to be a key parameter in the fate of bone tissue engineering scaffold. Finally, all these findings are translated into a particular clinical situation, revision knee arthroplasty, using the developed mathematical model. The thesis is presented in the form of an introduction, five chapters in article format, a conclusion, and an appendix. In chapter 1, it is shown that bone formation inside scaffold follows a diffusion phenomenon. An analytical formulation for bone formation is developed, which has only three parameters: C, the final bone volume fraction, α, the so-called scaffold osteoconduction coefficient, and h, the so-called peri-scaffold osteoinduction coefficient. The three parameters are estimated by identifying the model with in vivo data of polymeric scaffolds implanted in the femoral condyle of rats. in vivo data are obtained by longitudinal micro-CT scanning of the animals. Having identified the three parameters, the model is validated by predicting the course of bone formation in two previously published in vivo studies. We find the predicted values to be consistent with the experimental values. This model allows us to spatially and temporally predict the bone formation in scaffolds with only three physically relevant parameters. In chapter 2, whether the preparation of implantation site has an impact on bone formation inside scaffolds is investigated. For this purpose, two different drilling techniques are used to create a hole in distal femur of rats: a wood drill bit and a metal drill bit. The bone volume, bone mineral density, and callus formation are assessed non-invasively using micro-CT scanning at several time points after implantation. It is found that when a wood drill bit is used, the bone formation in the scaffold is accelerated by three weeks compared to when a metal drill bit is used. The bone-scaffold interfaces made by the two techniques are then compared. It is found that metal drill bit makes an interface which is less permeable compared to the one made by wood drill bit. In chapter 3, it is investigated if a cyclic loading can be used as stimulatory signal for bone formation in a bone scaffold. PLA/β-TCP scaffolds are implanted in both distal femoral epiphyses of rats. The right knee is stimulated (10 N, 4 Hz, 5 min) five times in total, once every two days, starting from the third day after implantation while the left knee serves as control. Finite element study of the in vivo experiment shows that the strain applied to the scaffold is similar to physiological strain. Using longitudinal micro-CT, all knees are scanned five times after the implantation and the related bone parameters of the newly formed bone are measured. The results show that mechanical stimulation had two effects on bone volume in the scaffold: an initial decrease at week 2, and a long-term increase in the rate of bone formation by 28%. At week 13, the bone volume is significantly higher in the loaded scaffolds. In chapter 4, the role of cyclic loading on mechanical properties of the bone-scaffold construct is evaluated. Following a similar loading protocol to the third chapter, the effect of loading on the stiffness of bone-scaffold construct is evaluated using micro-finite element modeling. It is found that loading increases the stiffness by 60% at 35 weeks. The increase of stiffness is correlated with an 18% increase of bone volume fraction in the loaded scaffold compared to the control scaffold. Bone volume fraction in the scaffold is compiled from two processes, namely bone formation and bone resorption. Using longitudinal micro-CT scanning and particular image registration techniques, it is observed that mechanical stimulation increases the rate of bone formation during the period 4-10 weeks, and inhibits bone resorption during the period 9-18 weeks post-operatively. Upon further investigation of the shapes of formation/resorption sites, it is found that loading mostly has an effect on the surface occupied by, rather than the thickness of those sites. For the first time, this study quantifies the effect of mechanical stimulation on bone formation/resorption inside a scaffold. In chapter 5, the possibility of using a PLA/β-TCP scaffold in revision knee arthroplasty in an epiphyseal defect is investigated. The mathematical model developed in chapter 1 enables us to translate the in vivo data from chapter 4 to a targeted clinical application. A finite element model of the revision knee arthroplasty is developed to estimate the mechanical stimulus applied on the scaffold. The course of bone formation in the scaffold is then estimated spatially and temporally. Using a multi-scale finite element analysis, the Young's moduli of the scaffolds are calculated. It is found that mechanical stimulation increases the bone volume fraction by 1.5-fold, and the stiffness by 2.5-fold in the scaffold, after three years. In the Appendix, following the methods developed in chapter 5, the effect of bone-scaffold interface preparation is investigated for a cystic defect in revision knee arthroplasty. The numerical model shows that by using a wood drill bit, bone volume fraction and stiffness of the bone-scaffold construct increase permanently after 3 years

Constructing 'homeland' and 'target' cities in the 'war on terror'.

No abstract available for this item